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SPEAKER VERIFICATION METHOD 

Field of the Invention 

The present invention is related to the field of speech recognition 
systems and more particularly to a speaker verification method. 

Background of the Invention 

Speech recognition and speaker verification use similar analysis 
tools to achieve its goals. An input utterance is first processed to 
determine its essential characteristics. Typically, input utterances are 
converted to cepstrum coefficients. A cepstrum is an inverse Fourier 
transform of the log power spectrum. In a training phase the cepstrum 
coefficients are saved to form code books for specific utterances. For 
instance, a code book might have codes for the numeral zero through 
nine. In speech recognition, an input utterance is compared to the codes 
(training utterances) in the code book to determine which is most 
similar. In speech recognition the code is a generalized representation of 
many people's way of forming an utterance (e.g., "zero"). In speaker 
verification the codes represent the individual characteristics of the 
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speaker and the verification system tries determine if a person's code is 
more similar to an input utterance, than an impostor code. As a result 
the codes in a speaker verification system emphasis individual 
characteristics, while in a speech recognition system the codes generalize 

5 over many individual speakers. Speaker verification has potential 

applications in a number of voice activated systems, such as banking 
over the telephone. Unfortunately, present speaker verification systems 
have not proven reliable enough for these applications. 

Thus there exists a need for an improved speaker verification 

10 system. 




Summary of the Tnvention 

An improved speaker verification method consist of the 
following steps: (1) generating a code book covering a number of 
speakers having a number of training utterances for each of the 
speakers; (2) receiving a number of test utterances from a speaker; 
(3) comparing each of the test utterances to each of the training 
utterances for the speaker to form a number of decisions, one 
decision for each of the number of test utterances; (4) weighting 
each of the decisions to form a number of weighted decisions; and 
(5) combining the plurality of weighted decisions to form a 
verification decision. 
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Brief Description of the Drawings 

FIG. 1 is a block diagram of an embodiment of a speaker 
verification system; 

5 FIG. 2 is a flow chart of an embodiment of the steps used to form a 

speaker verification decision; 

FIG. 3 is a flow chart of an embodiment of the steps used to form a 
code book for a speaker verification decision; 

FIG. 4 is a flow chart of an embodiment o. the steps used to form a 
10 speaker verification decision; 

FIG. 5 is a schematic diagram of a dial-up service that incorporates 
a speaker verification method; 

FIG. 6 is a flow chart of an embodiment of the steps used in a dial- 
up service; and 

15 FIG. 7 is a flow chart of an embodiment of the steps used in a dial- 

up service. 
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DPtniled Description of the Drawings 

Several improvements in speaker verification methods are 
described and then a dial-up service that can incorporate these 

5 improvements is explained. FIG. 1 is a block diagram of an embodiment 
of a speaker verification system 10. It is important to note that the 
speaker verification system can be physically implemented in a number 
of ways. For instance, the system can be implemented as software in a 
general purpose computer connected to a microphone; or the system can 

10 be implemented as firmware in a general purpose microprocessor 
connected to memory and a microphone; or the system can be 
implemented using a Digital Signal Processor (DSP), a controller, a 
memory, and a microphone controlled by the appropriate software. Note 
that since the process can be performed using software in a computer, 

15 then a computer readable storage medium containing computer readable 
instructions can be used to implement the speaker verification method. 
These various system architectures are apparent to those skilled in the 
art and the particular system architecture selected will depend on the 
application. 

20 a microphone 12 receives an input speech and converts the sound 

waves to an electrical signal. A feature extractor 14 analyzes the 
electrical signal and extracts key features of the speech. For instance, 
the feature extractor first digitizes the electrical signal. A cepstrum of 
the digitized signal is then performed to determine the cepstrum 

25 coefficients. In another embodiment, a linear predictive analysis is used 
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to find the linear predictive coding (LPC) coefficients. Other feature 
extraction techniques are also possible. 

A switch 16 is shown attached to the feature extractor 14. This 
switch 16 represents that a different path is used in the training phase 
than in the. verification phase. In the training phase the cepstrum 
coefficients are analyzed by a code book generator 18. The output of the 
code book generator 18 is stored in the code book 20. In one 
embodiment, the code book generator 18 compares samples of the same 
utterance from the same speaker to form a generalized representation of 
the utterance for that person. This generalized representation is a 
training utterance in the code book. The training utterance represents 
the generalized cepstrum coefficients of a user speaking the number 
"one" as an example. A training utterance could also be a part of speech, 
a phoneme, or a number like "twenty one" or any other segment of 
15 speech. In addition to the registered users samples, utterances are 

taken from a group of non-users. These utterances are used to form a 
composite that represents an impostor code having a plurality of 
impostor utterances. 

In one embodiment, the code book generator 18 determines 
20 whether the speaker (users and non-users) is male or female. The male 
training utterances (male group) are aggregated to determining a male 
variance vector. The female training utterances (female group) are 
aggregated to determine a female variance vector. These gender specific 
variance vectors will be used when calculating a weighted Euclidean 
25 distance (measure of closeness) in the verification phase. 
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In the verification phase the switch 16 connects the feature 
extractor 14 to the comparator 22. The comparator 22 performs a 
mathematical analysis of the closeness between a test utterance from a 
speaker with a training utterance stored in the code book 20 and 
between the test utterance and an impostor utterance. In one 
embodiment, a test utterance such as a spoken "one" is compared with 
the "one" training utterance and the "one" impostor utterance in the code 
book 20. The comparator 22 determines a measure of closeness 
between the "one" training utterance, the "one" test utterance and the 
"one" impostor utterance. When the test utterance is closer to the 
training utterance than the impostor utterance, the speaker is verified as 
the true speaker. Otherwise the speaker is determined to be an 
impostor. In one embodiment, the measure of closeness is a modified 
weighted Euclidean distance. The modification in one embodiment 
involves using a generalized variance vector instead of an individual 
variance vector for each of the registered users. In another 
embodiment, a male variance vector is used for male speakers and a 
female variance vector is used for a female speaker. 

A decision weighting and combining system 24 uses the measure 
of closeness to determine if the test utterance is closest to the training 
utterance or the impostor utterance. When the test utterance is closer to 
the training utterance than impostor utterance, a verified decision is 
made. When the test utterance is not closer to the training utterance 
than the impostor utterance, an un-verified decision is made. These are 
preliminary decisions. Usually, the speaker is required to speak several 
utterances (e.g., "one", "three", "five", "twenty one"). A decision is made 
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for each of these test utterances. Each of the plurality of decisions is 
weighted and combined to form the verification decision. 

The decisions are weighted because not all utterances provide 
equal reliability. For instance, "one" could provide a much more reliable 

5 decision than "eight". As a result, a more accurate verification decision 
can be formed by first weighting the decisions based on the underlying 
utterance. Two weighting methods can be used. One weighting method 
uses a historical approach. Sample utterances are compared to the 
training utterances to determine a probability of false alarm P FA (speaker 

10 is not impostor but the decision is impostor) and a probability of miss P M 
(speaker is impostor but the decision is true speaker). The P FA and P M 
are probability of errors. These probability of errors are used to weight 
each decision. In one embodiment the weighting factors (weight) are 
described by the equation below: 
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a, = log 1- P ^ Decision is Verified (Tr:e Speaker) 

20 a s = log P Mi Decision is Not Verified (Impostor) 

1 - P FAi 
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When the sum of the weighted decisions is greater than zero, then 
the verification decision is a true speaker. Otherwise the verification 
decision is an impostor. 
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The other method of weighting the decisions is based on an 
immediate evaluation of the quality of the decision. In one embodiment, 
this is calculated by using a Chi-Squared detector. The decisions are 
then weighted on the confidence determined by the Chi-Squared 
5 detector. In another embodiment, a large sample approximation is used. 
Thus if the test statistics are t, find b such that c 2 (b) = t. Then a decision 
is an impostor if it exceeds the 1-a quantile of the c 2 distribution. 

One weighting scheme is shown below: 

10 

1.5, if b > c accept 
1.0, if 1-a <b <c accept 
-1.0, if c reject <b < 1-a 
-1.25, if b < c reject 

15 

When the sum of the weighted decisions is greater than zero, then 
the verification decision is a true speaker. When the sum of the 
weighted decision is less than or equal to zero, the decision is an 
impostor. 

20 In another embodiment, the feature extractor 14 segments the 

speech signal into voiced sounds and unvoiced sounds. Voiced sounds 
generally include vowels, while most other sounds are unvoiced. The 
unvoiced sounds are discarded before the cepstrum coefficients are 
calculated in both the training phase and the verification phase. 
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These techniques of weighting the decisions, using gender 
dependent cepstrums and only using voiced sounds can be combined or 
used separately in a speaker verification system. 

FTG. 2 is a flow chart of an embodiment of the steps used to form a 
speaker verification decision. The process starts, at step 40, by 
generating a code book at step 42. The code book has a plurality of 
training utterances for each the plurality of speakers (registered users, 
plurality of people) and a plurality of impostor utterances. The training 
utterances in one embodiment are the cepstrum coefficients for a 
particular user speaking a particular utterance (e.g., "one}. The training 
utterances are generated by a user speaking the utterances. The 
cepstrum coefficients of each of the utterances are determined to form 
the training utterances. In one embodiment a speaker is asked to repeat 
the utterance and a generalization of the two utterances is saved as the 
training utterance. In another embodiment both utterances are saved as 

training utterances. 

In one embodiment, a data base of male speakers is used to 
determine a male variance vector and a data base of female speakers is 
used to determine a female variance vector. In another embodiment, 
the data bases of male and female speakers are used to form a male 
impostor code book and a female impostor code book. The gender 
specific variance vectors are stored in the code book. At step 44, a 
plurality of test utterances (input set of utterances) from a speaker are 
received. In one embodiment the cepstrum coefficients of the test 
25 utterances are calculated. Each of the plurality of test utterances are 

compared to the plurality of training utterances for the speaker at step 
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46. Based on the comparison, a plurality of decision are formed, one for 
each of the plurality of training utterances. In one embodiment, the 
comparison is determined by a Euclidean weighted distance between the 
test utterance and the training utterance and between the test utterance 
5 and an impostor utterance. In another embodiment, the Euclidean 
weighted distance is calculated with the male variance vector if the 
speaker is a male or the female variance vector if the speaker is a 
female. Each of the plurality of decisions are weighted to form a 
plurality of weighted decisions at step 48. The weighting can be based 

10 on historical error rates for the utterance or based on a confidence level 
(confidence measure) of the decision for the utterance. The plurality of 
weighted decisions are combined at step 50. In one embodi:::-;it the 
step of combining involves summing the weighted decisions. A 
verification decision is then made based on the combined weighted 

15 decisions at step 52, ending the process at step 54. In one embodiment 
if the sum is greater than zero, the verification decision is the speaker is 
a true speaker, otherwise the speaker is an impostor. 

FIG. 3 is a flow chart of an embodiment of the steps used to form a 
code book for a speaker verification decision. The process starts, at step 

20 70, by receiving an input utterance at step 72. In one embodiment, the 
input utterances are then segmented into a voiced sounds and an 
unvoiced sounds at step 74. The cepstrum coefficients are then 
calculated using the voiced sounds at step 76. The coefficients are 
stored as a training utterance for the speaker at step 78. The process 

25 then returns to step 72 for the next input utterance, until all the training 
utterances have been stored in the code book. 
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FIG. 4 is a flow chart of an embodiment of the steps used to form a 
speaker verification decision. The process starts, at step 100, by 
receiving input utterances at step 102. Next, it is determined if the 
speaker is male or female at step 104. In a speaker verification 
application, the speaker purports to be someone in particular. If the 
person purports to be someone that is a male, then the speaker is 
assumed to be male even if the speaker is a female. The input 
utterances are then segmented into a voiced sounds and an unvoiced 
sounds at step 106. Features (e.g., cepstrum coefficients) are extracted 
from the voiced sounds to form the test utterances, at step 108. At step 
110, the weighted Euclidean distance (WED) is calculated using either a 
generalized male variance vector if the purported speaker is a male. 
When the purported speaker is a female, the female variance vector is 
used. The WED is calculated between the test utterance and the trainins 
utterance for the speaker and the test utterance and the male (or female 
if appropriate) impostor utterance. A decision is formed for each test 
utterance based on the WED at step 112. The decisions are then 
weighted based on a confidence level (measure of confidence) 
determined using a Chi-squared detector at step 114. The weighted 
decisions are summed at step 116. A verification decision is made based 
on the sum of the weighted decisions at step 118. 

Using the speaker verification decisions discussed above results in 
an improved speaker verification system, that is more reliable than 
present techniques. 

A dial-up service that uses a speaker verification method as 
described above is shown in FIG. 5. The dial-up service is shown as a 
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banking service. A user dials a service number on their telephone 150. 
The public switched telephone network (PSTN) 152 then connects the 
user's phone 150 with a dial-up service computer 154 at a bank 156. 
The dial-up service need not be located within a bank. The service will 
be explained in conjunction with the flow chart shown in FIG. 6. The 
process starts, at step 170, by dialing a service number (communication 
service address, number) at step 172. The user (requester) is then 
prompted by the computer 154 to speak a plurality of digits (access 
code, plurality of numbers, access number) to form a first utterance at 
step 174. The digits are recognized using speaker independent voice 
recognition at step 176. When the user has used the dial-up service 
previously, verifying the user based on the first utterance at step 178. 
When the user is verified as a true speaker at step 178, allowing access 
to the dial-up service at step 180. When the user cannot be verified, 
requesting the user input a personal identification number (PIN) at step 
182. The PIN can be entered by the user either by speaking the PIN or 
by enter the PIN on a keypad. At step 184 it is determined if the PIN is 
valid. When the PIN is not valid, the user is denied access at step 186. 
When the PIN is valid the user is allowed access to the service at step 
180. Using the above method the dial-up service uses a speaker 
verification system as a PIN option, but does not deny access to the user 
if it cannot verify the user. 

FIG. 7 is a flow chart of another embodiment of the steps used in a 
dial-up service. The process starts, step 200, by the user speaking an 
access code to form a plurality of utterances at step 202. At step 204 it 
is determined if the user has previously accessed the service. When the 
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user has previously used the service, the speaker verification system 
attempts to verify the user (identity) at step 206. When the speaker 
verification system can verify the user, the user is allowed access to the 
system at step 208. When the system cannot verify the user, a PIN is 
requested at step 210. Note the user can either speak the PIN or enter 
the PIN on a keypad. At step 212 it is determined if the PIN is valid. 
When the PIN is not valid the user is denied access at step 214. When 
the PIN is valid, the user is allowed access at step 208. 

When the user has not previously accessed the communication 
service at step 204, the user is requested to enter a PIN at step 216. At 
step 218 it is determined if the PIN is valid at step 218. When the PIN 
is not valid, denying access to the service at step 220. When the PIN is 
valid the user is asked to speak the access code a second time to form a 
second utterance (plurality of second utterances) at step 222. The 
similarity between the first utterance (step 202) and the second 
utterance is compared to a threshold at step 224. In one embodiment 
the similarity is calculated using a weighted Euclidean distance. When 
the similarity is less than or equal to the threshold, the user is asked to 
speak the access code again at step 222. In this case the second and 
third utterances would be compared for the required similarity. In 
practice, the user would not be required to repeat the access code at step 
222 more than once or twice and the system would then allow the user 
access. When the similarity is greater than the threshold, storing a 
combination of the two utterances as at step 226. In another 
embodiment both utterances are stored as reference utterances. Next 
access to the service is allowed at step 208. The reference utterance 
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(plurulity of reference utterances, reference voiced sounds) is used to 
verify the user the next time they access the service. Note that the 
speaker verification part of the access to the dial-up service in one 
embodiment uses all the techniques discussed for a verification process. 
In another embodiment the verification process only uses one of the 
speaker verification techniques. Finally, in another embodiment the 
access number has a predetermined digit that is selected from a first set 
of digits (predefined set of digits) if the user is a male. When the user is 
a female, the predetermined digit is selected from a second set of digits. 
This allows the system to determine if the user is suppose to be a male 
or a female. Based on this information, the male variance vector or 
female variance vector is used in the speaker verification process. 

Thus there has been described an improved speaker verification 
method and a service that takes advantage of the speaker verification 
method. While the invention has been described in conjunction with 
specific embodiments thereof, it is evident that many alterations, 
modifications, and variations will be apparent to those skilled in the art 
in light of the foregoing description. Accordingly, it is intended to 
embrace all such alterations, modifications, and variations in the 
appended claims. 



